Facial images retrieval system

ABSTRACT

A facial images retrieval system is provided. The facial images retrieval system is adapted to receive an initial textual description of a facial image to perform an initial facial image search that obtains a plurality of facial images based on the textual description. The facial images retrieval system then receives a selection of the first and second facial images that are relatively close to a desired facial image to perform a further facial image search to obtain another facial image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/729,194 filed on Sep. 10, 2018, the entire contents of which arehereby incorporated by reference herein.

BACKGROUND

The inventors herein have recognized a need for improved facial imageretrieval system.

In particular, the inventors herein have recognized a need for a systemthat receives an initial textual description of a facial image toperform an initial facial image search to obtain a plurality of facialimages based on the textual description, and then to receive a selectionof at least first and second facial images that are relatively close toa desired facial image to perform further facial image searching toobtain another facial image.

Further, the inventors herein have recognized a need for a system thatreceives an initial textual description of a facial image to perform aninitial facial image search to obtain a plurality of facial images basedon the textual description, and then to allow modification of one of thefacial images to obtain a modified facial image that is closer to adesired image, to perform further facial image searching based on themodified facial image to obtain another facial image.

Further, the inventors herein have recognized a need for a system thatreceives an initial textual description of a facial image to perform aninitial facial image search to obtain a plurality of facial images basedon the textual description, and then to receive a modified textualdescription and a selection of one of the facial images to performfurther facial image searching to obtain another facial image.

SUMMARY

A facial images retrieval system in accordance with an exemplaryembodiment is provided. The facial images retrieval system includes adisplay device and a computer operably coupled to the display device.The computer receives a first textual description of a facial image. Thecomputer determines a first similarity score associated with the firsttextual description and a first facial image, and a second similarityscore associated with the first textual description and a second facialimage. The computer instructs the display device to display the firstand second facial images thereon. The computer receives a user selectionof the first and second facial images to perform further facial imagesearching. The computer determines a third similarity score associatedwith the first facial image and a third facial image, and a fourthsimilarity score associated with the second facial image and the thirdfacial image. The similarity scorer module calculates a weighted averageof the third and fourth similarity scores to determine a finalsimilarity score of the third facial image. The computer instructs thedisplay device to display the third facial image and the finalsimilarity score thereon.

A facial images retrieval system in accordance with another exemplaryembodiment is provided. The facial images retrieval system includes adisplay device and a computer operably coupled to the display device.The computer receives a first textual description of a facial image. Thecomputer determines a first similarity score associated with the firsttextual description and a first facial image, and a second similarityscore associated with the first textual description and a second facialimage. The computer instructs the display device to display the firstand second facial images thereon. The computer receives a userinstruction to modify soft-biometric attributes of the first facialimage to obtain a first modified facial image to perform further facialimage searching. The computer determines a third similarity scoreassociated with the first modified facial image and a third facialimage. The computer instructs the display device to display the thirdfacial image and the final similarity score thereon.

A facial images retrieval system in accordance with another exemplaryembodiment is provided. The facial images retrieval system includes adisplay device and a computer operably coupled to the display device.The computer receives a first textual description of a facial image. Thecomputer determines a first similarity score associated with the firsttextual description and a first facial image, and a second similarityscore associated with the first textual description and a second facialimage. The computer instructs the display device to display the firstand second facial images thereon. The computer receives a user selectionof the first facial image and the textual description to perform furtherfacial image searching. The computer determines a third similarity scoreassociated with the first facial image and a third facial image, and afourth similarity score associated with a second textual description andthe third facial image. The similarity scorer module calculates aweighted average of the third and fourth similarity scores to determinea final similarity score of the third facial image. The computerinstructs the display device to display the third facial image and thefinal similarity score thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a facial images retrieval system inaccordance with an exemplary embodiment;

FIG. 2 is a block diagram of a computer in the facial images retrievalsystem of FIG. 1 having an input graphical user interface, apre-processing module, a natural language processing module, asimilarity scorer module, a first computer vision computational neuralnetwork, and a second computer vision computational neural network;

FIG. 3 is a schematic of a first facial image and a master featurevector descriptor that describes attributes of the first facial image;

FIG. 4 is a schematic of a second facial image and a master featurevector descriptor that describes attributes of the second facial image;

FIG. 5 is a schematic of a third facial image and a master featurevector descriptor that describes attributes of the third facial image;

FIG. 6 is a schematic of a fourth facial image and a master featurevector descriptor that describes attributes of the fourth facial image;

FIG. 7 is a schematic of a graphical user interface utilized by thesystem of FIG. 1 for implementing a first method for searching forfacial images;

FIG. 8 is another schematic of the graphical user interface of FIG. 7;

FIG. 9 is another schematic of the graphical user interface of FIG. 7;

FIG. 10 is another schematic of the graphical user interface of FIG. 7;

FIGS. 11-13 are flowcharts of a first method for searching for facialimages utilizing the facial images retrieval system of FIG. 1;

FIG. 14 is a schematic of the graphical user interface utilized by thesystem of FIG. 1 for implementing a second method for searching forfacial images;

FIG. 15 is another schematic of the graphical user interface of FIG. 14;

FIG. 16 is another schematic of the graphical user interface of FIG. 14;

FIG. 17 is another schematic of the graphical user interface of FIG. 14;

FIGS. 18-20 are flowcharts of a second method for searching for facialimages utilizing the facial images retrieval system of FIG. 1;

FIG. 21 is a schematic of the graphical user interface utilized by thesystem of FIG. 1 for implementing a third method for searching forfacial images;

FIG. 22 is another schematic of the graphical user interface of FIG. 21;

FIG. 23 is another schematic of the graphical user interface of FIG. 21;

FIG. 24 is another schematic of the graphical user interface of FIG. 21;

FIGS. 25-27 are flowcharts of a third method for searching for facialimages utilizing the facial images retrieval system of FIG. 1; and

FIG. 28 is a block diagram illustrating that the NLP module converts atextual description to a textual feature vector.

DETAILED DESCRIPTION

Referring to FIGS. 1 and 2, a facial images retrieval system 20 inaccordance with an exemplary embodiment is provided. The facial imagesretrieval system 20 includes a computer 30, an image database 40, anembedding database 45, an input device 50, and a display device 60.

The computer 30 is operably coupled to the input device 50, the displaydevice 60, the image database 40, and the embedding database 45. Thecomputer 30 includes an input graphical user (GUI) interface 100, apre-processing module 102, a natural language processing (NLP) module104, a similarity scorer module 106, a first computer visioncomputational neural network 108, a second computer vision computationalneural network 110, which will be described in greater detail below.

The input device 50 is provided to receive user selections forcontrolling operation of the computer 30. The display device 60 isprovided to display the GUI 100 and facial images in response toinstructions received from the computer 30.

Referring to FIGS. 1 and 3-6, the image database 40 is provided to storea plurality of facial images therein. In particular, the image database40 stores the facial image 200 (shown in FIG. 3), the facial image 202(shown in FIG. 4), the facial image 204 (shown in FIG. 5), and thefacial image 206 (shown in FIG. 6). In an exemplary embodiment, theimage database 40 stores a plurality of additional facial imagestherein.

Referring to FIGS. 1 and 3-6, the embedding database 45 is provided tostore a plurality of master feature vector descriptors therein. Eachmaster feature vector descriptor is associated with a facial image inthe image database 40 and have numbers that describe soft-biometricattributes and features of the facial image. In particular, theembedding database 45 stores a master feature vector descriptor 300(shown in FIG. 3) having numbers that describe attributes and featuresof the facial image 200. For example, the master feature vectordescriptor 300 includes a number 0.9 which corresponds to an age rangeof 26-35 and the facial features for the woman identified in the facialimage 200. Further, the embedding database 45 stores a master featurevector descriptor 302 (shown in FIG. 4) having numbers that describeattributes and features of the facial image 202. Further, the embeddingdatabase 45 stores a master feature vector descriptor 304 (shown in FIG.5) having numbers that describe attributes and features of the facialimage 204. Further, the embedding database 45 stores a master featurevector descriptor 306 (shown in FIG. 6) having numbers that describeattributes and features of the facial image 206.

Referring to FIG. 28, the NLP module 100 receives a textual descriptionof a desired facial image and converts the textual description to atextual feature vector which has numbers that describes attributes inthe desired facial image with respect to the textual description. Thetextual feature vector is compared to master feature vector descriptorsassociated with facial images to determine similarity scores.

An advantage of the facial images retrieval system 20 is that the system20 is adapted to receive an initial textual description of a facialimage to perform an initial facial image search to obtain a plurality offacial images based on the textual description, and then to receive aselection of first and second facial images that are relatively close toa desired facial image to perform a further facial image search toobtain another facial image.

Another advantage of the facial images retrieval system 20 is that thesystem 20 is adapted to receive an initial textual description of afacial image to perform an initial facial image search to obtain aplurality of facial images based on the textual description, and then toreceive user instructions to modify one of the facial images to obtain amodified mage that is closer to a desired image, and to perform anotherfacial image search based on the modified image to obtain another facialimage.

Another advantage of the facial images retrieval system 20 is that thesystem 20 is adapted to receive an initial textual description of afacial image to perform an initial facial image search that obtains aplurality of facial images based on the textual description, and then toreceive another textual description and a selection of one of the facialimages to perform a further facial image search to obtain another facialimage.

For purposes of understanding, a few technical terms used herein willnow be explained.

The term “feature” is a recognizable pattern that is consistentlypresent in a facial image. An exemplary feature is a hair style.

The term “attribute” is an aggregate of a set of features determined bya plurality of features in a facial image. Exemplary attributes includeage, ethnicity, bald, gender, hair color, face shape, skin tone, skinvalue, eye shape, eye size, eye color, eye character, forehead, nosebridge, lip shape, lip size, lip symmetry, mustache, and beard.

Referring to FIGS. 7-10, a description of the input GUI 100 forimplementing a first method for searching for facial images utilizingthe facial images retrieval system 20 will be explained. During thefirst method, the input GUI 100 receives an initial textual descriptionof a facial image to perform an initial facial image search to obtain aplurality of facial images based on the textual description, and thenreceives a selection of the first and second facial images (that arerelatively close to a desired facial image) to perform a further facialimage search to obtain another facial image.

Referring to FIG. 7, the input GUI 100 includes a text display region400, a submission display region 402, and an image display region 404.The input device 50 is utilized to make the user selections and inputteddata in the input GUI 100.

The text display region 400 includes a text input box 410, a textselection checkbox 412, and an edit command button 414. The text inputbox 410 allows a user to input a textual description for a desired imageutilizing the input device 50. For example, an exemplary textualdescription recites: “I'd like an image of a young blonde with a smile.She should have an oval face with small eyes.” The text selectioncheckbox 412 allows a user to indicate that a textual description isbeing input by the user and is to be used for a facial image search. Theedit command button 414 allows the user to edit the textual descriptionin the text input box 410.

The submission display region 402 includes a text selected message 416and a submit command button 418. The text selected message 416 indicatesthat the text selection checkbox 412 has been selected indicating atextual description is being utilized for the facial image search. Whenthe submit command button 418 is selected by the user, the computer 30performs a facial image search and displays the search results as shownin FIG. 8.

Referring to FIG. 8, the image display region 404 includes a facialimage 200 found during the facial image search that has one of thehighest similarity scores. The image display region 404 further includesa hair color adjustment slider 480, an eye size adjustment slider 482,an age adjustment slider 484, a similarly score box 486, and an imageselection checkbox 488 that are associated with the facial image 200.The hair color adjustment slider 480 allows a user to adjust the haircolor of the facial image 200 for further facial image searching. Theeye size adjustment slider 482 allows the user to adjust the eye size ofthe facial image 200 for further facial image searching. The ageadjustment slider 484 allows the user to adjust the age of the face inthe facial image 200 for further facial image searching. The similarlyscore box 486 indicates a similarity score between the facial image 200and the textual description within the text input box 410. The imageselection checkbox 488 allows a user to select whether to use the facialimage 200 for further facial image searching.

The image display region 404 further includes a facial image 204 foundduring the facial image search that has one of the highest similarityscores. The image display region 404 further includes a hair coloradjustment slider 580, an eye size adjustment slider 582, an ageadjustment slider 584, a similarly score box 586, and an image selectioncheckbox 588 that are associated with the facial image 204. The haircolor adjustment slider 580 allows a user to adjust the hair color ofthe facial image 204 for further facial image searching. The eye sizeadjustment slider 582 allows the user to adjust the eye size of thefacial image 204 for further facial image searching. The age adjustmentslider 584 allows the user to adjust the age of the face in the facialimage 204 for further facial image searching. The similarly score box586 indicates a similarity score between the facial image 204 and thetextual description within the text input box 410. The image selectioncheckbox 588 allows a user to select whether to use the facial image 204for further facial image searching.

Referring to FIG. 9, the image selection checkbox 488 has been selectedwhich indicates that the facial image 200 is to be utilized for afurther facial image search. Further, the image selection checkbox 588has been selected which indicates that the facial image 204 is to beutilized for further facial image search.

The submission display region 402 includes a weighting value adjustmentslider 630, a weighting value adjustment slider 632, and a submitcommand button 418. The weighting value adjustment slider 630 allows theuser to select the weighting value that will be assigned to the facialimage 200 for determining a weighted average similarity score associatedwith a new facial image found in a facial image search. The weightingvalue adjustment slider 630 allows the user to select the weightingvalue that will be assigned to the facial image 204 for determining theweighted average similarity score associated with a new facial imagefound in a facial image search. The submit command button 418 allows auser to instruct the computer 30 to perform the facial image searchbased on the facial image 200 and the facial image 204.

Referring to FIGS. 1, 2 and 10, after the facial image search isperformed, the image display region 404 includes the facial image 206and the similarity score box 640. The computer 30 performs the facialimage search based on the facial images 200, 204 (shown in FIG. 9) andretrieves the facial image 206 based on the attributes in the facialimages 200, 204. Further, the similarly score box 640 indicates aweighted average similarity score of the facial image 206.

Referring to FIGS. 1, 2 and 11-13, a flowchart of a first method forsearching for facial images utilizing the facial images retrieval system20 will be explained. For purposes of simplicity, the first method willbe explained utilizing three facial images. However, it should beunderstood that the first method could be implemented with numerousfacial images and determine numerous similarity scores to determine thefacial images with the highest similarity scores and to display suchfacial images.

At step 520, the computer 30 executes the input graphical user interface(GUI) 100, the pre-processing module 102, the natural languageprocessing (NLP) module 104, the first computer vision computationalneural network 108, the second computer vision computational neuralnetwork 110, and the similarity scorer module 106. After step 520, themethod advances to step 524.

At step 524, the image database 40 stores facial images 200, 204, 206therein. After step 524, the method advances to step 526.

At step 526, the input GUI 100 on the display device 60 receives a firsttextual description of a facial image and sends the first textualdescription to the pre-processing module 102. After step 526, the methodadvances to step 528.

At step 528, the pre-processing module 102 performs tokenization of thefirst textual description to obtain a first list of textual words thatare sent to the NLP module 104. After step 528, the method advances tostep 530.

At step 530, the NLP module 104 generates a first textual feature vectordescriptor 180 (shown in FIG. 180) based on the first list of textualwords. The first textual feature vector descriptor 180 is sent to thesimilarity scorer module 106. At step 530, the method advances to step532.

At step 532, the similarity scorer module 106 determines a firstsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a first master feature vector descriptor 300(shown in FIG. 3) associated with the facial image 200, utilizing afollowing equation: first similarity score=f(first textual featurevector descriptor, first master feature vector descriptor), wherein fcorresponds to a similarity function. The first master feature vectordescriptor 300 is generated by the second computer vision computationalneural network 110 utilizing the facial image 200 and is stored in anembedding database 45. After step 532, the method advances to step 540.

At step 540, the similarity scorer module 106 determines a secondsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a second master feature vector descriptor 304associated with the facial image 204, utilizing a following equation:second similarity score=f(first textual feature vector descriptor,second master feature vector descriptor), wherein f corresponds to asimilarity function. The second master feature vector descriptor 304 isgenerated by the second computer vision computational neural network 110utilizing the second facial image and is stored in the embeddingdatabase 45. After step 540, the method advances to step 542.

At step 542, the computer 30 instructs the display device 60 to displaythe facial images 200, 204 and the first and second similarity scoresthereon. After step 542, the method advances to step 544.

At step 544, the input GUI 100 receives a user selection for the facialimages 200, 204 to perform further facial image searches, and sends thefacial images 200, 204 to the pre-processing module 102. The input GUI100 further receives a user selection for first and second weightingvalues associated with the facial images 200, 204, respectively. Afterstep 544, the method advances to step 546.

At step 546, the pre-processing module 102 normalizes and aligns thefacial image 200 to obtain at a first pre-processed facial image, andnormalizes and aligns the facial image 204 to obtain at a secondpre-processed facial image sends the first and second pre-processedfacial images to the first computer vision computational neural network108. After step 546, the method advances to step 560.

At step 560, the first computer vision computational neural network 108generates third and fourth master feature vector descriptors based onthe first and second pre-processed facial images, respectively, andsends the third and fourth master feature vector descriptors to thesimilarity scorer module 106. After step 560, the method advances tostep 562.

At step 562, the similarity scorer module 106 determines a thirdsimilarity score between the third master feature vector descriptor anda fifth master feature vector descriptor 306 associated with a facialimage 206, utilizing a following equation: third similarityscore=f(third master feature vector descriptor, a fifth master featurevector descriptor), wherein f corresponds to a similarity function. Thefifth master feature vector descriptor is generated by the firstcomputer vision computational neural network 108 and is stored in theembedding database 45. After step 562, the method advances to step 564.

At step 564, the similarity scorer module 106 determines a fourthsimilarity score between the fourth master feature vector descriptor andthe fifth master feature vector descriptor associated with the facialimage 206 utilizing a following equation: fourth similarityscore=f(fourth master feature vector descriptor, fifth master featurevector descriptor), wherein f corresponds to a similarity function.After step 564, method advances to step 566.

At step 566, the similarity scorer module 106 calculates a weightedaverage of at least the third and fourth similarity scores to determinea final similarity score of the facial image 206, utilizing thefollowing equation: final similarity score=(first weighting value×thirdsimilarity score)+(second weighting value×fourth similarity score)/2.After step 566, the method advances to step 568.

At step 568, the computer 30 instructs the display device 60 to displaythe facial image 206 and the final similarity score thereon.

Referring to FIGS. 14-17, a description of the input GUI 100 forimplementing a second method for searching for facial images utilizingthe facial images retrieval system 20 will be explained. During thesecond method, the input GUI 100 receives an initial textual descriptionof a facial image to perform an initial facial image search to obtain aplurality of facial images based on the textual description, and thenreceives user instructions to modify one of the facial images to obtaina modified facial image that is closer to a desired image, and toperform a further facial image search based on the modified facial imageto obtain another facial image.

Referring to FIG. 14, the text display region 400 includes a text inputbox 410, a text selection checkbox 412, and an edit command button 414.The text input box 410 indicates the user entered the textualdescription for a desired image as: “I'd like an image of a young blondewith a smile. She should have an oval face with small eyes.” When theuser selects the submit command button 418, the computer 30 performs thefacial image search.

Referring to FIG. 15, the image display region 404 displays the facialimages 200, 204 which were found in the facial image search.

Referring to FIG. 16, the user has utilized the eye size adjustmentslider 482 to adjust the eye size in the facial image 200 (shown in FIG.15) to obtain a modified facial image 204 (shown in FIG. 16). Further,the image selection checkbox 488 has been selected to indicate that themodified facial image 204 will be utilized for further facial imagesearching. When the user selects the submit command button 418, thecomputer 30 performs the refined facial image search.

Referring to FIG. 17, the image display region 404 displays the facialimage 206 which was found in the refined facial image search based onthe modified facial image 204. Further, the similarity score box 640indicates a similarity score for the facial image 206.

Referring to FIGS. 1, 2 and 18-20, a flowchart of a second method forsearching for facial images utilizing the facial images retrieval system20 will be explained. For purposes of simplicity, the second method willbe explained utilizing three facial images. However, it should beunderstood that the second method could be implemented with numerousfacial images and determine numerous similarity scores to determine thefacial images with the highest similarity scores and to display suchfacial images.

At step 700, the computer 30 executes the input graphical user interface(GUI) 100, the pre-processing module 102, the natural languageprocessing (NLP) module, the first computer vision computational neuralnetwork 108, the second computer vision computational neural network110, and the similarity scorer module 106. After step 700, the methodadvances to step 702.

At step 702, the image database 40 stores first, second, and thirdfacial images 200, 204, 206 therein. After step 702, the method advancesto step 704.

At step 704, the input GUI 100 on the display device 60 receives a firsttextual description of a facial image and sends the first textualdescription to the pre-processing module 102. After step 704, the methodadvances to step 706.

At step 706, the pre-processing module 102 performs tokenization of thefirst textual description to obtain a first list of textual words thatare sent to the NLP module 104. After step 706, the method advances tostep 708.

At step 708, the NLP module 104 generates a first textual feature vectordescriptor 180 (shown in FIG. 28) based on the first list of textualwords. The first textual feature vector descriptor 180 is sent to thesimilarity scorer module 106. After step 708, the method advances tostep 710.

At step 710, the similarity scorer module 106 determines a firstsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a first master feature vector descriptor 300(shown in FIG. 3) associated with the facial image 200 utilizing afollowing equation: first similarity score=f(first textual featurevector descriptor, first master feature vector descriptor), wherein fcorresponds to a similarity function. The first master feature vectordescriptor 300 is generated by the second computer vision computationalneural network 110 and is stored in an embedding database 45. After step710, the method advances to step 720.

At step 720, the similarity scorer module 106 determines a secondsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a second master feature vector descriptor 304(shown in FIG. 5) associated with the facial image 204, utilizing afollowing equation: second similarity score=f(first textual featurevector descriptor, second master feature vector descriptor), wherein fcorresponds to a similarity function. The second master feature vectordescriptor 304 is generated by the second computer vision computationalneural network 110 and is stored in an embedding database 45. After step720, the method advances to step 722.

At step 722, the computer 30 instructs the display device 60 to displaythe facial images 200, 204 and the first and second similarity scoresthereon. After step 722, the method advances to step 724.

At step 724, the input GUI 100 receives user instructions to modifysoft-biometric attributes of the facial image 200 (shown in FIG. 15) toobtain a first modified facial image 204 (shown in FIG. 16), and sendsthe first modified facial image 204 to the pre-processing module 102.After step 724, the method advances to step 726.

At step 726, the pre-processing module 102 normalizes and aligns thefirst modified facial image 204 (shown in FIG. 16) to obtain at a firstpre-processed facial image, and sends the first pre-processed facialimage to the first computer vision computational neural network 108.After step 726, method advances to step 728.

At step 728, the first computer vision computational neural network 108generates a third master feature vector descriptor based on the firstpre-processed facial image, and sends the third master feature vectordescriptor to the similarity scorer module 106. After step 728, themethod advances to step 730.

At step 730, the similarity scorer module 106 determines a thirdsimilarity score between the third master feature vector descriptor anda fourth master feature vector descriptor 306 (shown in FIG. 6)associated with a facial image 206, utilizing a following equation:third similarity score=f(third master feature vector descriptor, fourthmaster feature vector descriptor), wherein f corresponds to a similarityfunction. The fourth master feature vector descriptor is generated bythe first computer vision computational neural network 108. After step730, the method advances to step 732.

At step 732, the computer 30 instructs the display device 60 to displaythe facial image 206 and the third similarity score thereon.

Referring to FIGS. 21-24, a description of the input GUI 100 forimplementing a third method for searching for facial images utilizingthe facial images retrieval system 20 will be explained. During thethird method, the input GUI 100, the input GUI 100 receives an initialtextual description of a facial image to perform an initial facial imagesearch to obtain a plurality of facial images based on the textualdescription, and then receives a modified textual description and aselection of one of the facial images to perform a further facial imagesearch to obtain another facial image.

Referring to FIG. 21, the text display region 400 includes a text inputbox 410, a text selection checkbox 412, and an edit command button 414.The text input box 410 indicates the user entered the textualdescription for a desired image as: “I'd like an image of a young blondewith a smile. She should have an oval face with small eyes.” When theuser selects the submit command button 418, the computer 30 performs thefacial image search.

Referring to FIG. 22, the image display region 404 displays the facialimages 200, 204 which were found in the facial image search.

Referring to FIG. 23, the user has selected the image selection checkbox488 for performing further facial image searching. The user has furtherselected the text selection checkbox 412 indicating that the modifiedtextual description in the text input box 410 should also be utilizedfor further facial image searching. The weighting value adjustmentslider 804 allows the user to select the weighting value that will beassigned to the modified textual description in textual input box 410for determining a weighted average similarity score. The weighting valueslider 806 allows the user to select the weighting value to be assignedto be facial image 200 for determining the weighted average similarityscore. When the user selects the submit command button 418, the computer30 performs the refined facial image search.

Referring to FIG. 24, the image display region 404 displays the facialimage 206 which was found in the facial image search based on the facialimage 200 and the modified textual description. Further, the similarityscore box 640 indicates a similarity score for the facial image 206.

Referring to FIGS. 1, 2 and 25-27, a flowchart of a third method forsearching for facial images utilizing the facial images retrieval system20 will be explained. For purposes of simplicity, the third method willbe explained utilizing three facial images. However, it should beunderstood that the third method could be implemented with numerousfacial images and determine numerous similarity scores to determine thefacial images with the highest similarity scores and to display suchfacial images.

At step 900, the computer 30 executes the input graphical user interface(GUI), the pre-processing module 102, the natural language processing(NLP) module, the first computer vision computational neural network108, the second computer vision computational neural network 110, andthe similarity scorer module 106. After step 900, method advances tostep 902.

At step 902, the image database 40 stores first, second, and thirdfacial images 200, 204, 206 therein. After step 902, method advances tostep 904.

At step 904, the input GUI 100 on the display device 60 receives a firsttextual description of a facial image and sends the first textualdescription to the pre-processing module 102. After step 904, the methodadvances to step 906.

At step 906, the pre-processing module 102 performs tokenization of thefirst textual description to obtain a first list of textual words thatare sent to the NLP module 104. After step 906, the method advances tostep 908.

At step 908, the NLP module 104 generates a first textual feature vectordescriptor 180 (shown in FIG. 28) based on the first list of textualwords. The first textual feature vector descriptor 180 is sent to thesimilarity scorer module 106. After step 908, the method advances tostep 910.

At step 910, the similarity scorer module 106 determines a firstsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a first master feature vector descriptor 300(shown in FIG. 3) associated with the facial image 200, utilizing afollowing equation: first similarity score=f(first textual featurevector descriptor, first master feature vector descriptor), wherein fcorresponds to a similarity function. The first master feature vectordescriptor 300 is generated by the second computer vision computationalneural network 110 utilizing the first facial image and is stored in anembedding database 45. After step 910, the method advances to step 920.

At step 920, the similarity scorer module 106 determines a secondsimilarity score between the first textual feature vector descriptor 180(shown in FIG. 28) and a second master feature vector descriptor 304(shown in FIG. 5) associated with the facial image 204, utilizing afollowing equation: second similarity score=f(first textual featurevector descriptor, second master feature vector descriptor), wherein fcorresponds to a similarity function. The second master feature vectordescriptor 304 is generated by the second computer vision computationalneural network 110 utilizing the second facial image and is stored inthe embedding database 45. After step 920, the method advances to step922.

At step 922, the computer 30 instructs the display device 60 to displaythe facial images 200, 204 and the first and second similarity scoresthereon. After step 922, the method advances to step 924.

At step 924, the input GUI 100 receives a user selection for the facialimage 200 and a second textual description of the facial image toperform further facial image searches, and sends the facial image 200and the second textual description to the pre-processing module 102. Theinput GUI 100 further receives a user selection of a first weightingvalue for the facial image 200, and a second weighting value associatedwith the second textual feature vector descriptor, respectively. Afterstep 924, the method advances to step 926.

At step 926, the pre-processing module 102 normalizes and aligns thefacial image 200 to obtain at a first pre-processed facial image, andsends the first pre-processed facial images to the first computer visioncomputational neural network 108. After step 926, the method advances tostep 928.

At step 928, the first computer vision computational neural network 108generates a third master feature vector descriptor based on the firstpre-processed facial image and sends the third master feature vectordescriptor to the similarity scorer module 106. After step 928, themethod advances to step 930.

At step 930, the pre-processing module 102 performs tokenization of thesecond textual description to obtain a second list of textual words thatare sent to the NLP module 104. After step 930, the method advances tostep 932.

At step 932, the NLP module 104 generates a second textual featurevector descriptor based on the second list of textual words. The secondtextual feature vector descriptor is sent to the similarity scorermodule 106. After step 932, the method advances to step 934.

At step 934, the similarity scorer module 106 determines a thirdsimilarity score between the third master feature vector descriptor anda fourth master feature vector descriptor 306 (shown in FIG. 6)associated with the facial image 206, utilizing a following equation:third similarity score=f(third master feature vector descriptor, afourth master feature vector descriptor), wherein f corresponds to asimilarity function. The fourth master feature vector descriptor 306 isgenerated by the first computer vision computational neural network 108and is stored in the embedding database 45. After step 934, the methodadvances to step 936.

At step 936, the similarity scorer module 106 determines a fourthsimilarity score between the second textual feature vector descriptorand the fourth master feature vector descriptor 306 (shown in FIG. 6)associated with the facial image 206, utilizing a following equation:fourth similarity score=f(second textual feature vector descriptor,fourth master feature vector descriptor), wherein f corresponds to asimilarity function. After step 936, the method advances to step 938.

At step 938, the similarity scorer module 106 calculates a weightedaverage of at least the third and fourth similarity scores to determinea final similarity score of the third facial image, utilizing thefollowing equation: final similarity score=(first weighting value×thirdsimilarity score)+(second weighting value×fourth similarity score)/2.After step 938, the method advances to step 940.

At step 940, the computer 30 instructs the display device 60 to displaythe facial image 206 and the final similarity score thereon.

While the claimed invention has been described in detail in connectionwith only a limited number of embodiments, it should be readilyunderstood that the invention is not limited to such disclosedembodiments. Rather, the claimed invention can be modified toincorporate any number of variations, alterations, substitutions orequivalent arrangements not heretofore described, but which arecommensurate with the spirit and scope of the invention. Additionally,while various embodiments of the claimed invention have been described,it is to be understood that aspects of the invention may include onlysome of the described embodiments. Accordingly, the claimed invention isnot to be seen as limited by the foregoing description.

What is claimed is:
 1. An image retrieval system, comprising: a displaydevice; a computer operably coupled to the display device, the computerhaving an input graphical user interface (GUI), a pre-processing module,a natural language processing (NLP) module, and a similarity scorermodule; the input GUI receiving a first textual description of a facialimage; the pre-processing module performing tokenization of the firsttextual description to obtain a first list of textual words describingthe facial image; the NLP module generating a first textual featurevector descriptor based on the first list of textual words describingthe facial image, the first textual feature vector descriptor being sentto the similarity scorer module; the similarity scorer moduledetermining a first similarity score between the first textual featurevector descriptor and a first master feature vector descriptorassociated with a first facial image, the first master feature vectordescriptor being generated by a first computer vision computationalneural network utilizing the first facial image; and the similarityscorer module determining a second similarity score between the firsttextual feature vector descriptor and a second master feature vectordescriptor associated with a second facial image; the second masterfeature vector descriptor being generated by the first computer visioncomputational neural network utilizing the second facial image; thecomputer instructing the display device to display the first and secondfacial images thereon at a same time; the computer receiving a userselection of the first and second facial images and first and secondweighting values that are associated with the first and second facialimages, respectively, to perform further facial image searching; thecomputer determining a third similarity score associated with the firstfacial image and a third facial image, and a fourth similarity scoreassociated with the second facial image and the third facial image; thesimilarity scorer module calculating a weighted average of the third andfourth similarity scores utilizing the first and second weighting valuesto determine a final similarity score of the third facial image; and thecomputer instructing the display device to display the third facialimage and the final similarity score thereon.
 2. The facial imagesretrieval system of claim 1, wherein the computer instructing thedisplay device to display the first and second facial images thereoncomprises: the computer instructing the display device to display thefirst and second facial images and the first and second similarityscores thereon.
 3. The facial images retrieval system of claim 1,wherein the computer determining the third similarity score associatedwith the first facial image and the third facial image, and the fourthsimilarity score associated with the second facial image and the thirdfacial image comprises: the pre-processing module normalizing andaligning the first facial image to obtain at a first pre-processedfacial image, and normalizing and aligning the second facial image toobtain at a second pre-processed facial image sending the first andsecond pre-processed facial images to a second computer visioncomputational neural network; the second computer vision computationalneural network generating third and fourth master feature vectordescriptors based on the first and second pre-processed facial images,respectively, and sending the third and fourth master feature vectordescriptors to the similarity scorer module; the similarity scorermodule determining the third similarity score between the third masterfeature vector descriptor and a fifth master feature vector descriptorassociated with the third facial image, the fifth master feature vectordescriptor being generated by the second computer vision computationalneural network; the similarity scorer module determining the fourthsimilarity score between the fourth master feature vector descriptor andthe fifth master feature vector descriptor associated with the thirdfacial image.
 4. The facial images retrieval system of claim 1, whereinthe similarity scorer module calculating the weighted average of thethird and fourth similarity scores to determine the final similarityscore of the third facial image, comprises the similarity scorer modulecalculating the weighted average utilizing a following equation:final similarity score=(the first weighting value×the third similarityscore)+(the second weighting value×the fourth similarity score)/2.
 5. Animage retrieval system, comprising: a display device; a computeroperably coupled to the display device, the computer having an inputgraphical user interface (GUI), a pre-processing module, a naturallanguage processing (NLP) module, and a similarity scorer module; theinput GUI receiving a first textual description of a facial image; thepre-processing module performing tokenization of the first textualdescription to obtain a first list of textual words describing thefacial image; the NLP module generating a first textual feature vectordescriptor based on the first list of textual words describing thefacial image, the first textual feature vector descriptor being sent tothe similarity scorer module; the similarity scorer module determining afirst similarity score between the first textual feature vectordescriptor and a first master feature vector descriptor associated witha first facial image, the first master feature vector descriptor beinggenerated by a first computer vision computational neural networkutilizing the first facial image; the similarity scorer moduledetermining a second similarity score between the first textual featurevector descriptor and a second master feature vector descriptorassociated with a second facial image; the second master feature vectordescriptor being generated by the first computer vision computationalneural network utilizing the second facial image; the computerinstructing the display device to display the first and second facialimages thereon at a same time; the computer receiving a user instructionto modify soft-biometric attributes of the first facial image to obtaina first modified facial image to perform further facial image searching;the computer determining a third similarity score associated with thefirst modified facial image and a third facial image; and the computerinstructing the display device to display the third facial image and thethird similarity score thereon.
 6. The facial images retrieval system ofclaim 5, wherein the computer instructing the display device to displaythe first and second facial images thereon comprises: the computerinstructing the display device to display the first and second facialimages and the first and second similarity scores thereon.
 7. The facialimages retrieval system of claim 5, wherein the computer receiving theuser instruction to modify soft-biometric attributes of the first facialimage to obtain the first modified facial image to perform furtherfacial image searching, comprises: the input GUI receiving the userinstruction to modify soft-biometric attributes of the first facialimage to obtain the first modified facial image, and sending the firstmodified facial image to the pre-processing module.
 8. The facial imagesretrieval system of claim 7, wherein the computer determining the thirdsimilarity score associated with the first modified facial image and thethird facial image, comprises the pre-processing module normalizing andaligning the first modified facial image to obtain at a firstpre-processed facial image, and sending the first pre-processed facialimage to a second computer vision computational neural network; thesecond computer vision computational neural network generating a thirdmaster feature vector descriptor based on the first pre-processed facialimage, and sending the third master feature vector descriptor to thesimilarity scorer module; the similarity scorer module determining athird similarity score between the third master feature vectordescriptor and a fourth master feature vector descriptor associated witha third facial image, the fourth master feature vector descriptor beinggenerated by the second computer vision computational neural network. 9.An image retrieval system, comprising: a display device; a computeroperably coupled to the display device, the computer having an inputgraphical user interface (GUI), a pre-processing module, a naturallanguage processing (NLP) module, and a similarity scorer module; theinput GUI receiving a first textual description of a facial image; thepre-processing module performing tokenization of the first textualdescription to obtain a first list of textual words describing thefacial image that are sent to the NLP module; the NLP module generatinga first textual feature vector descriptor based on the first list oftextual words describing the facial image, the first textual featurevector descriptor being sent to the similarity scorer module; thesimilarity scorer module determining a first similarity score betweenthe first textual feature vector descriptor and a first master featurevector descriptor associated with a first facial image, the first masterfeature vector descriptor being generated by a first computer visioncomputational neural network utilizing the first facial image; and thesimilarity scorer module determining the second similarity score betweenthe first textual feature vector descriptor and a second master featurevector descriptor associated with a second facial image; the secondmaster feature vector descriptor being generated by the first computervision computational neural network utilizing the second facial image;the computer instructing the display device to display the first andsecond facial images thereon at a same time; the computer receiving auser selection of the first facial image and a second textualdescription and first and second weighting values that are associatedwith the first facial image and the second textual description,respectively, to perform further facial image searching; the computerdetermining a third similarity score associated with the first facialimage and a third facial image, and a fourth similarity score associatedwith a second textual description and the third facial image; thesimilarity scorer module calculating a weighted average of the third andfourth similarity scores utilizing the first and second weighting valuesto determine a final similarity score of the third facial image; and thecomputer instructing the display device to display the third facialimage and the final similarity score thereon.
 10. The facial imagesretrieval system of claim 9, wherein the computer instructing thedisplay device to display the first and second facial images thereoncomprises: the computer instructing the display device to display thefirst and second facial images and the first and second similarityscores thereon.
 11. The facial images retrieval system of claim 9,wherein the computer receiving the user selection of the first facialimage and the second textual description to perform further facial imagesearching comprises: the input GUI receiving the user selection for thefirst facial image and the second textual description of the facialimage to perform further facial image searches, and sending the firstfacial image and the second textual description to the pre-processingmodule, the input GUI further receiving the user selection of the firstweighting value for the first facial image, and the second weightingvalue associated with a second textual feature vector associated withthe second textual description, respectively.
 12. The facial imagesretrieval system of claim 11, wherein the computer determining the thirdsimilarity score associated with the first facial image and the thirdfacial image, and the fourth similarity score associated with the secondtextual description and the third facial image comprises: thepre-processing module normalizing and aligning the first facial image toobtain at a first pre-processed facial image and sends the firstpre-processed facial image to a second computer vision computationalneural network; the second computer vision computational neural networkgenerating a third master feature vector descriptor based on the firstpre-processed facial image, and sends the third master feature vectordescriptor to the similarity scorer module; the similarity scorer moduledetermining the third similarity score between the third master featurevector descriptor and a fourth master feature vector descriptorassociated with the third facial image, the fourth master feature vectordescriptor being generated by the second computer vision computationalneural network; the pre-processing module performing tokenization of thesecond textual description to obtain a second list of textual words thatare sent to the NLP module; the NLP module generating the second textualfeature vector descriptor based on the second list of textual words, thesecond textual feature vector descriptor being sent to the similarityscorer module; and the similarity scorer module determines a fourthsimilarity score between the second textual feature vector descriptorand the fourth master feature vector descriptor associated with thethird facial image.
 13. The facial images retrieval system of claim 12,wherein the similarity scorer module calculating the weighted average ofthe third and fourth similarity scores to determine the final similarityscore of the third facial image, comprises the similarity scorer modulecalculating the weighted average utilizing a following equation:the final similarity score=(the first weighting value×third similarityscore)+(the second weighting value×the fourth similarity score)/2.